Estimating Prediction Error: Cross-Validation vs. Accumulated Prediction Error

The Estimation of Prediction Error: Covariance Penalties and Cross-Validation

Having constructed a data-based estimation rule, perhaps a logistic regression or a classification tree, the statistician would like to know its performance as a predictor of future cases. There are two main theories concerning prediction error: (1) penalty methods such as Cp, Akaike’s information criterion, and Stein’s unbiased risk estimate that depend on the covariance between data points an...

Cross-Validation and the Bootstrap: Estimating the Error Rate of a Prediction Rule

A training set of data has been used to construct a rule for predicting future responses. What is the error rate of this rule? The traditional answer to this question is given by cross-validation. The cross-validation estimate of prediction error is nearly unbiased, but can be highly variable. This article discusses bootstrap estimates of prediction error, which can be thought of as smoothed ve...

Decomposition of Prediction Error

Surprise beyond prediction error

Surprise drives learning. Various neural "prediction error" signals are believed to underpin surprise-based reinforcement learning. Here, we report a surprise signal that reflects reinforcement learning but is neither un/signed reward prediction error (RPE) nor un/signed state prediction error (SPE). To exclude these alternatives, we measured surprise responses in the absence of RPE and account...

Grammatical error prediction

In this thesis, we investigate methods for automatic detection, and to some extent correction , of grammatical errors. The evaluation is based on manual error annotation in the Cambridge Learner Corpus ((((), and automatic or semi-automatic annotation of error corpora is one possible application, but the methods are also applicable in other settings, for instance to give learners feedback on th...